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ABSTRACT KEYWORDS 
Although newly passed federal regulations mandate the provision Adolescence; evidence- 
of family-based aftercare supports for adolescents served in ther- based practices; family; 


apeutic residential care, very little research has been conducted to transition 
determine best practice. This efficacy and replication study com- 
pares the outcomes of youths and caregivers randomly assigned to 
On the Way Home (OTWH; n = 98) or traditional aftercare supports 
(n = 89) following discharge from therapeutic residential care (TRC). 
Findings were mixed. At posttest (12-months) no significant differ- 
ences were found between groups on indicators of placement 
stability and school involvement, however, significant differences 
were found between groups on several indicators of caregiver 
empowerment and self-efficacy, with caregivers in OTWH reporting 
greater levels of self-efficacy and empowerment across the 
domains of family and community. At follow-up (21-months), mod- 
erate to large differences were found between groups on indicators 
of placement stability and school involvement, with odds ratios 
indicating youths in OTWH were 2 and 3 times more likely to be 
engaged in school and living in the community, respectively. 
Implications, limitations, and future research are discussed. 


The 1970's ushered in an important and significant shift in therapeutic resi- 
dential care (TRC) regarding the perceived roles of families in youths’ short 
and long-term wellbeing (Leichtman, 2006). Once marginalized in program- 
ming, and long identified as a shortcoming of many TRCs in the past (e.g., 
Barth, 2005; Jenson & Whittaker, 1987; Leichtman, 2006), comprehensive 
literature reviews and efficacy evaluations revealed the engagement and 
empowerment of parents/caregivers in the child’s treatment to be an impor- 
tant component in treatment efficacy as well as in the potential for long-term 
maintenance of therapeutic gains (Guerts, Boddy, Noom, & Knorth, 2012; 
Jenson & Whittaker, 1987; Lakin, Brambila, & Sigda, 2004; Landsman, Groza, 
Tyler, & Malone, 2001). Despite these findings, the adoption of specified and 
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efficacious strategies for promoting family/caregiver engagement and empow- 
erment in TRC programming has been slow (Leichtman, 2006). 

In the United States, efforts to bolster the roles of families in TRCs is most 
recently reflected in the passage of the 2018 Family First Prevention Services Act 
(H.R. 5456; Public Law No: 115-123) which reforms child welfare financing 
streams to provide additional services for families at-risk of entering the child 
welfare system. Primary aims of the bill are to (a) prevent children from entering 
into the foster care system through the provision of parent, mental health, and 
substance use supports and (b) incentivize states to reduce placement of children 
in congregate care settings such as TRCs by increasing the availability of family 
supports including intensive in-home prevention services (Torres & Mathur, 
2018). However, in recognition that TRCs may be the most appropriate and 
least restrictive setting for some children and youths, provisions in the legislation 
also mandate that among other requirements (e.g., trauma-informed treatment 
model, licensed staff and accredited programming), congregate care agencies must 
facilitate and document family involvement during the child’s placement and 
provide a minimum of 6-months post-discharge family-based aftercare support to 
the child and family following reunification (Torres & Mathur, 2018). Although 
this legislation further highlights the importance of supporting systems involved 
families, these requirements pose challenges to service providers seeking to 
identify and implement efficacious, evidence-supported programs to meet these 
provisions. This new requirement for TRCs to provide family-based aftercare 
programming is especially difficult as evidence-based models for aftercare support 
are significantly lacking for reunifying youths (Author, 2013, 2013b; Author; 2014; 
Daniel, Goldston, Harris, Kelley, & Palmes, 2004; Whittaker, 2000). 

In 2007, the Institute for Education Sciences at the U.S. Department of 
Education funded the development and _ evaluation (Author, 2007; 
R324B070034) and later efficacy and replication (Author, 2012a; R324A120260) 
studies of On the Way Home (OTWH), a family-centred aftercare program 
designed to promote family stability and youth school engagement following 
youth reintegration into the home and community school settings following 
placements in TRC (Author, 2012b). The program was a result of findings from 
a series of studies on the comprehensive mental health, behavioral health, aca- 
demic, language, and physical health needs of children and youths in TRCs as well 
as short and long-term efficacy studies on outcomes of youths served in these 
placements (Author, 2017). These foundational studies laid the groundwork for 
determining gaps in service delivery and highlighted the lack of evidence-based 
aftercare supports for school-aged children in TRCs who would be returning 
home prior to aging out of the system (Author, 2017). Primary goals of OTWH 
were to provide reunifying families and youths with evidence-supported programs 
that promote the maintenance of gains made while in care, teach new skills to 
families and youths to promote placement permanency and implement strategies 
to foster school engagement and high school graduation (Author, 2013). 
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On the Way Home modified and combined three evidence-supported inter- 
ventions; Check & Connect (C&C; Christenson, Evelo, Sinclair, & Thurlow, 1997), 
Common Sense Parenting (CSP; Burke & Herron, 1996), and a homework inter- 
vention (Author, 2012b; see detailed description below). Designed with the 
primary objectives of keeping families reunified and youth engaged in school, 
a central tenet of OTWH is the promotion of placement stability through family 
empowerment (i.e., the degree to which parents perceive their ability to, and level 
of comfort with, navigating services on their child’s behalf; Bode et al., 2016) and 
caregiver self-efficacy (ie., the level of knowledge and degree of confidence in 
effectively performing caregiver roles such as providing support and managing 
family challenges; Bandura, Caparara, Barbaranelli, Regalia, & Scabini, 2011). 
Broadly studied in child development literature, these constructs have been 
identified as influential factors in child school-refusal and academic achievement 
(e.g., Kim & Bryan, 2017; Lynch, 2002; Steca, Bassi, Caprara, & Fave, 2011); child 
engagement in, and benefit from, mental health services (e.g., Taub, Tighe, & 
Burchard, 2001); and improved parenting and family functioning (e.g., Rominov, 
Giallo, & Whelan, 2016; Steca et al., 2011). 

In our first efficacy randomized controlled trial (RCT) evaluation of OTWH 
with 82 middle and high school aged youths at-risk and their families, study 
discharge data revealed that youths randomly assigned to the control condition 
(n = 39; i.e., services-as-usual; SAU) were three times more likely to return to 
care and five times more likely to discontinue enrollment in the community 
school at the end of services one year post family reunification from TRC 
(Author, 2013). While preliminary, these promising findings demonstrated 
that the provision of clearly defined and targeted aftercare programming that 
support the youth, school, and family can help to mitigate the challenges often 
demonstrated during reunification such as school failure and placement instabil- 
ity (Author, 2013). Based on the results of the first RCT, OTWH was placed on 
the California Evidence-Based Clearinghouse (CEBC, 2019b) as a promising 
practice and the only aftercare intervention on the CEBC that included both 
family and school supports for middle and high school-aged youth who were 
departing residential programs. Notable limitations of this study, however, were 
the small sample size, lack of follow-up data collected post aftercare program 
completion, and minimal information regarding the impact of OTWH on key 
family skills which may be associated with long-term success (e.g., family self- 
efficacy and caregiver empowerment; Author, 2013). 

To address the primary limitations of the first OTWH RCT, a follow-up efficacy 
and replication study was conducted to better understand the program’s impact 
on these key caregiver factors and on youth posttest and follow-up indicators of 
school engagement and placement stability. Specifically, this study sought to 
evaluate the effects of OTWH on family empowerment and caregiver self-efficacy 
at program discharge (i.e., 12-months following reunification) as well as program 
effects on school involvement (i.e., graduated or enrolled in school) and placement 
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stability (i.e., living in the community) at program discharge and at 9 months 
following study completion (i.e., 21-months following reunification). 


Method 
Procedure 


Study procedures were approved by the University of Nebraska-Lincoln, school 
district, and agency Institutional Review Boards. Eligible participants included 
youths who discharged from the participating residential agencies between 
2012-2017 and met the following criteria: (a) enrolled in grades 8-12; (b) returning 
to home, school, and community settings within a 90-mile radius of the residential 
agencies or university; and (c) identified with (ie., receiving special education 
services via an Individualized Education Program; IEP) or at-risk (i-e., Diagnostic 
Interview Schedule for Children [DISC-IV] mental health diagnosis on file; 
Shaffer, Fisher, Lucas, Dulcan, & Schwab-Stone, 2000) of a high-incidence dis- 
ability. For planned discharges (i.e., youths who met all program requirements and 
goals; n = 236), youth/caregiver dyads were identified, contacted, and recruited via 
the participating residential agencies approximately 6 weeks prior to discharge. For 
unplanned discharges (i.e., youths who left care prior to reaching program goals 
due to running away, parent withdraw, or court-ordered withdraw; n = 15), youth/ 
caregiver dyads were approached upon discharge. Participation was voluntary; 
both the youth and caregiver were required to assent/consent. A total of 251 youth/ 
caregiver dyads met eligibility criteria and were approached for participation. We 
were unable to establish contact with 17 of those youth/families. Of the remaining 
234, 38 declined participation, 196 provided consent/assent, and 9 consented/ 
assented but declined after randomization to the treatment or control condition. 
Ninety-eight were randomly assigned to OTWH and 89 to SAU. For those 
assigned to OTWH, participants were assigned to one of five different interven- 
tionists (i.e., Family Consultants [FCs]; see Figure 1). Because only participants in 
OTWH were nested within interventionists (i-e., participants in the SAU were not 
nested), this was a partially nested RCT (Lohr, Schochet, & Sanders, 2014) rather 
than a typical RCT with randomization at the individual level. 


Participants 


Overall Sample 

The overall sample included 187 child-caregiver dyads of youths departing resi- 
dential group care settings that were licensed as group homes that used the Boys 
Town Family Home Program (Thompson & Daly, 2015) or Teaching-Family 
Model (Phillips, Phillips, Fixsen, & Wolf, 1974). For the youth participants, the 
majority were male (58.3%) with a mean age of 15.45 years (SD = 1.44). Sixty- 
one percent (n = 115) reported their race as white and 12.8% reported their 
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Figure 1. Partially nested RCT. 


ethnicity as Latino/a. Over 90% of youths (n = 172) had met agency goals and were 
successfully discharged from residential care. Nearly 60% (n = 111) had an 
individualized education program (IEP) in place upon returning to their com- 
munity school. Caregivers reported on average, youths had 1.91 (SD = 1.57) 
formal prior placements (Range = 0 to > 9) and 0.50 (SD = 0.96) informal prior 
placements (Range = 0 to 6). For caregiver participants, the majority were female 
(73.3%) with a mean age of 44.56 years (SD = 9.59). Sixty-nine percent (n = 129) 
reported their race as white and 6.4% reported their ethnicity as Latino/a. Nearly 
45% (n = 84) reported an annual income of less than $30,000, 58.8% reported 
having more than a high school degree, 63.6% were biological parents, and 32.6% 
were currently married. 


Analytic Samples 
Three different analytic samples were used in the effectiveness analyses based 
on the type of outcome measure and /or time point at which the data were 
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collected (ie., posttest caregiver outcomes, posttest school and community 
placement outcomes, and follow-up school and community placement out- 
comes). Because different outcomes were collected using different methods 
(e.g., caregiver report, family consultant report) or at different time points 
(e.g., posttest or follow-up), different patterns of missing data were observed 
for the different outcomes resulting in three different analytic samples. The 
analytic sample for posttest caregiver outcomes included 128 participants 
(50% in the OTWH condition), the analytic sample for posttest placement 
outcomes included 155 participants (50.32% in the OTWH condition), and 
the analytic sample for the follow-up placement outcomes included 132 
participants (53.79% in the OTWH condition). 

Table 1 reports the demographic variables for participants in the OTWH and 
SAU conditions, respectively, as well as the standardized differences between the 
two conditions for each analytic sample. Standardized differences were 
expressed as Hedges’ g estimates (Hedges, 1981) for continuous variables and 
as Cox d estimates (Cox, 1970) for binary variables. Both Hedges’ g and Cox 
d estimates are interpreted as the mean difference between conditions in stan- 
dard deviation units. Differences between the OTWH and SAU conditions are 
described below in the “Baseline Equivalence” section. 


Program Description and Study Design 


Youth/caregiver dyads were randomly assigned to OTWH or SAU following 
assent/consent. Participants in SAU received the traditional services available to 
all discharging youths including a departure planning meeting to discuss goals 
achieved while in care, continuing treatment recommendations, medical needs, 
and school information. Youths/caregivers were also provided with information 
on the Boys Town National Hotline, a no-cost 24-hour a day phone/text-based 
resource and counseling service accredited by the American Association of 
Suicidology and staffed by trained counselors. The hotline information is pro- 
vided to all discharging youths and is available in over 140 languages to all 
persons looking for expert advice on issues such as parenting, bullying, suicidal 
thoughts, family problems, and substance abuse (www.boystown.org). 
Participants assigned to OTWH received support for up to 12-months. 
Developed to promote caregiver empowerment and self-efficacy and youth school 
involvement, OTWH integrates three modified evidence supported and evidence- 
based interventions; Common Sense Parenting (Burke & Herron, 1996), Check & 
Connect (Christenson et al., 1997), and homework support (Author, 2012b). All 
three components were implemented simultaneously by the FCs in the family 
home, school, or community setting. Each component was selected to address 
primary risks commonly demonstrated during reunification including school 
failure or drop-out and family instability (Author, 2013; Author, 2012b). While 
not intended to be comprehensive wraparound support, FCs were trained to 
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provide crisis and outreach support (e.g., homelessness, behavioral/physical 
health, substance abuse) when circumstances affected the youth’s functioning at 
home or school. 


Common Sense Parenting 

(CSP; Burke & Herron, 1996). CSP was developed as a group, classroom-based, 6 
session, parent training program that is deemed supported by research evidence 
on the CEBC (2019a). To promote parenting self-efficacy and empowerment, 
FCs were trained to provide parents with a one-on-one in-home version of CSP 
developed for OTWH. When possible, the FCs began the six one-hour training 
sessions with the caregivers prior to reunification to establish a foundational set 
of skills. For unplanned discharges, CSP training started during the first week of 
services. The home-based CSP training consisted of six weekly sessions that 
provided caregivers with strategies to promote effective communication, prevent 
problem behaviors, improve discipline by correcting problem behaviors, and 
how to identify and encourage good behaviors and model self-control. Strategies 
were taught through direct instruction and caregivers worked through video 
examples, modeling, role-plays, and activities to demonstrate mastery. Unique 
to OTWH, the parenting skills taught one-on-one during the initial six CSP 
sessions were reinforced and reviewed weekly throughout the 12-month pro- 
gram and were individualized to address each family’s individual circumstances 
and youth behaviors to promote sustainability of the pro-social skills youth 
learned while in the residential program. 

Check & Connect (C&C; Christenson et al., 1997). C&C is an evidence-based 
dropout prevention, on the What Works Clearinghouse, deemed effective for 
helping students make progress and stay in school (Institute for Education 
Science, 2019). Originally developed and evaluated as a school-based interven- 
tion to prevent drop-out in at-risk populations (Christenson et al., 1997), we 
modified the components in C&C for individual use in OTWH. As designed, the 
“Check” component of C&C consists of monitoring malleable high-risk student 
behaviors by a school employee identified as the mentor. In OTWH, the FC 
monitored each youth’s high-risk behaviors (e.g., suspensions, detentions, fail- 
ing grades, tardies, and absences) via on-line school portals and entered the data 
into the on-line OTWH C&C database. Risk-criteria were set for each high-risk 
behavior (e.g., 3 or more tardies/month, 1 or more “D” or “F” grade) and alerts 
were emailed to the FC and FC supervisor (see description below) when the 
youth’s data indicated risk to prompt the FCs to begin the “Intensive” supports 
defined in “Connect”. 

The second component of C&C “Connect”, consisted of two interventions, 
“Basic” and “Intensive”. In “Basic”, the FCs promoted school engagement through 
weekly conversations with the youths and caregivers regarding educational goals 
and the youth’s school identity. Topics addressed included sharing general school- 
related information, homework concerns, participation in clubs/sports/activities, 
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goals for graduation, and strategies to address ongoing risks. In contrast, 
“Intensive” was designed to promote student engagement through addressing 
risk behaviors identified in the “Check” component. When a youth behavior 
flagged for risk (ie., met the preset risk criteria), the FC selected from the 
OTWH C&C intervention menu and met with the youth, caregiver, and when 
necessary, school personnel to begin implementation (Author, 2012b). The inter- 
vention menu outlined strategies for addressing maladaptive school behaviors in 
a systematic approach from least intensive (e.g., talk with student to determine 
reason, evaluate for behavioral patterns) to most (e.g., develop academic contract, 
identify online courses to address credit deficiencies, develop reward system). 
Interventions for both the “Basic” and “Intensive” C&C “Connect” components 
were entered and tracked in the OTWH C&C database and “Intensive” interven- 
tions continued until the weekly data reports no longer indicated risk. 

Homework intervention (Author, 2012b). The third component of OTWH 
involved a structured intervention designed to increase parent and youth 
perceptions of self-efficacy and empowerment in youth homework related 
tasks. There were four primary goals of the homework intervention: (1) 
improve homework completion; (2) decrease tension and friction related to 
homework completion; (3) improve home-school communication; and (4) 
build youth independence. Similar to C&C, the homework intervention 
involved “Basic” and “Enhanced” components. The “Basic” component was 
implemented with all youths and included the completion of a homework 
checklist and the development of a monitoring system. The homework checklist 
was a 13-item document that the youth and caregiver completed with the 
guidance of the FC. Items were designed to support families in establishing 
house rules and expectations regarding homework completion (e.g., when and 
where homework will be completed) as well as to identify any materials (e.g., 
calculator, computer) needed for completion. The homework checklist was 
completed within the first 6 weeks of intervention and was signed off by the 
youth, caregiver, and FC. The document was reviewed monthly with the youth/ 
caregiver and updated as necessary. For youths who continued to struggle with 
homework completion or who struggled in one or more academic area(s), FCs 
implemented the “Enhanced” component of the homework intervention. In 
“Enhanced”, the FC met with the caregiver, youth, and school to identify 
additional strategies (e.g., identifying a tutor, contacting a homework helpline, 
adding additional study period, establishing a positive or negative consequence 
system; Author, 2012b) to promote educational success. 


Family Consultants 

Minimal hiring requirements for FCs included a bachelor’s degree in a social 
service field and previous experience working with youth in the child welfare 
system. 
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FCs were trained on all program components during an intensive two-week 
training conducted by the study principal investigator, OTWH supervisor 
(licensed mental health professional [LMHP] employed by the residential 
agency), and agency staff. Approximately half of the training hours were spent 
on mastery of CSP, 15% on C&C, 10% on the homework strategy, and 25% on 
the OTWH program overview, roles, implementation procedures, and service 
and study related data collection and entry. 

FCs participated in multiple levels of weekly supervision. First, FCs attended 
weekly two-hour group supervision sessions to review (a) active individual 
cases; (b) eligibility and consent status of discharging youth still in care; and 
(c) research project procedures, documentation, and data collection. Second, 
weekly individual supervision sessions with the FC supervisor were held follow- 
ing group supervision (or more if needed) to address cases in crisis, service 
documentation, implementation fidelity, and when necessary, determine addi- 
tional training or skill development resources for the FC. Although service time 
varied by family need (e.g., crisis) and time (e.g., at reunification, start of 
school year), FCs were expected to make weekly contact with the youths and 
caregivers to work on family goals and objectives using the CSP materials, 
collect C&C risk indicator data and connect with youths and families weekly, 
and monitor homework completion. FCs carried caseloads of up to 15 youth/ 
caregiver dyads and spent an average of 104 hours (approximately 2 hours/ 
week) on each case over the 12-months of services. 


Measures 


Caregiver Empowerment 

Caregivers of participating youths completed the Family Empowerment Scale 
(FES; Koren, DeChillo, & Friesen, 1992) at posttest (ie., 12-months after youth 
discharge from residential care). The FES measures the empowerment of a parent 
or caregiver of a child with emotional disabilities. The 34 items are rated on 
a 5-point scale (1 = never to 5 = very often) with higher scores indicating greater 
levels of empowerment. Mean scores for the FES are calculated by summing scores 
for the subscale items and dividing by the number of items in each subscale: 
Family (12 items; e.g., I feel confident in my ability to help my child grow and 
develop, I have a good understanding of my child’s disorder), Service System (12 
items, e.g., I know what services my child needs, I am able to make good decisions 
about what services my child needs), and Community (10 items, e.g., I help other 
families get the services they need, I have ideas about the ideal service system for 
children). Studies of the FES reveal adequate psychometric properties (i.e., test- 
retest correlations from .77 to .85; internal consistency across subscales from .87 to 
.88; Koren et al., 1992). Subscale scores were used in the effectiveness analyses, and 
sample internal consistency for the subscale scores was .91, .93, and .87 for Family, 
Services, and Community, respectively. 
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Caregiver Self-efficacy 
Caregivers completed the 25-item Caregiver Self-Efficacy Scale (CSES; Boothroyd 
& Evans, 2000) along with the FES at posttest. The CSES measures caregiver 
perceptions regarding selective parenting skills and their perceived ability to care 
for their children. Items address behavior management (e.g., How comfortable 
are you with your ability to control your child’s behavior?), child advocacy (e.g., 
How comfortable are you with your ability to advocate for your child’s rights?), 
and management of school-related matters (e.g., How comfortable are you with 
your ability to participate in school activities with your child?). Rated on 
a 4-point Likert-type scale (1 = not very comfortable to 4 = very comfortable), 
the items comprise five subscales (i.e., Behavior Management, Advocacy, School 
Issues, Emotional Support, and Provider Issues). A total score can also be 
calculated which is the sum of the five subscale scores. For the CSES, higher 
scores indicate greater perceived caregiving self-efficacy. The CSES scores have 
documented adequate internal consistency with Cronbach alpha’s ranging from 
.61-.86 (Boothroyd & Evans, 1997). Only the total score was used in the effec- 
tiveness analyses, and the sample internal consistency for the total score was .93. 
Placement stability and school involvement. Youth placement stability and 
school involvement were collected by trained data collectors from the schools 
and families at posttest and follow-up (i.e., 21-months following reunification) 
using the School & Home Placement Change Questionnaire (SHPQ, Author, 
2013). The SHPQ was developed and evaluated in the previous RCT of 
OTWH and consists of two items (1) Where is the study child currently living? 
and (2) What is the study child’s current school placement status? Community 
placement was defined as living with a parent, primary caregiver, legal guardian, 
or independent living. School involvement was defined as maintaining enroll- 
ment in the community school setting, graduation, or working on a GED. 
Although youths who return to care or go to jail receive educational support, 
this involves a placement change, which prevents attendance at the home school. 


Demographic Covariates 

Demographic characteristics of caregivers and youths were included in the 
analyses as covariates, primarily to statistically adjust for differences between 
the OTWH and SAU conditions on factors which may be related to the study 
outcomes. Caregiver demographic covariates included age, income level, and 
the number of other children in the household. Caregiver age and number of 
youths in the household were grand mean-centred for the sample, and 
annual income level was coded as less than $30,000/year (1) vs. more than 
$30,000/year (0) (because income data were collected as categorical data). 
Youth demographic covariates included age, gender, and the length of time 
in residential care. Youth age and time in TRC were grand mean-centred for 
the sample. Youth gender was coded as male (1) or female (0). 
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Analysis Plan 


Attrition and Missing Data 

The What Works Clearinghouse (WWC) standards for RCTs (2017) were used 
to assess the overall attrition rate and the differential attrition rate based on 
observed data patterns. The WWC provides “cautious” and “optimistic” stan- 
dards for assessing the potential bias introduced by differential attrition. The 
optimistic guidelines should be applied when the underlying reason for attrition 
is believed to be exogenous - that is, the reason is unrelated to the intervention. 
On the other hand, the cautious guidelines should be applied when attrition is 
believed to be endogenous. For this RCT, we adopted the optimistic guidelines 
because, based on informal interviews with caregivers and reports by FCs, the 
majority of missing outcome data arose because families moved out of the study 
area, youth were incarcerated (or returned to residential care), graduated from 
high school, or there was a change in legal guardianship. None of these factors 
were considered endogenous to the study. If the combination of overall attrition 
and differential attrition rates fall within the cautious range, then the RCT is 
considered low attrition with limited risk of potential bias. When the attrition 
rates are considered to represent only limited potential bias, then the WWC 
supports the used = of complete case analysis (p. 38). 


Baseline Equivalence 

Per the WWC guidelines for RCTs, we evaluated baseline equivalence for the three 
analytic samples. No exogenous pretest measures were collected as part of this 
RCT; however, demographic and background variables were collected at baseline 
and compared for equivalence between conditions. As recommended by WWC, 
the magnitude of the differences between conditions was assessed rather than the 
statistical significance of the differences. To evaluate the magnitude of differences, 
effect sizes were computed. For continuous variables, Hedges’ g was computed, 
and for dichotomous variables, Cox’s d was computed. Cox’s d is an approxima- 
tion of Hedges’ g for dichotomous variables (Sanchez-Meca, Marin-Martinez, & 
Chacon-Moscoso, 2003). Variables that exhibited effect sizes less than 0.05 were 
considered equivalent, variables that demonstrated effect sizes between 0.05 and 
0.25 were considered for statistical adjustments in the effectiveness analyses, while 
effect sizes greater than 0.25 might represent non-equivalence between conditions. 


Program Effectiveness 

Because this RCT represents a partially nested RCT (PN-RCT; Lohr et al., 2014), 
the dependency introduced by nesting OTWH participants within intervention- 
ists needed to be accounted for in the analysis model (Bauer, Sterba, & Halfors, 
2008; Lohr et al., 2014). To this end, we used Hierarchical Linear Modeling v7 
software (HLM; Raudenbush, Bryk, & Congdon, 2013) to specify a fixed intercept, 
random slope multilevel model (MLM) adapted to account for the asymmetrical 
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variance structure of the partially nested RCT. The following model was used to 
analyze caregiver outcomes: 


Yi = Yoo + VigQOTWHy + Yo9(CG_ AGE; CG_ AGE.) 

+ Y3y)LOW — INCOME, + Y49(NUM-YOUTH; NUM-_YOUTH. ) 
+ Y5y¥ - MALE; + yo(¥Y- AGE; Y_AGE_)+y,..(LOS; LOS.) 
+ uyjOTWHi + rj 


Where Yj is the posttest score for individual i nested within intervention cluster j, 
Yoo is the mean posttest score for a caregiver in the SAU group conditional on other 
predictors (i.e., when all other predictors are zero), yio is the mean difference 
between the OTWH and SAU groups (conditional on the other predictors), y2o 
is the additive effect of grand mean centred caregiver age, 30 is the additive effect 
of dummy-coded family income level (less than $30,000/year), 49 is the additive 
effect of grand mean centred number of youths in the household, yso is the additive 
effect of dummy-coded youth gender (male), yeo is the additive effect of grand 
mean centred youth age, y7o is the additive effect of grand mean centred length of 
stay in TRC, u,; is the random slope variation for individuals in the OTWH 
condition, and rj is the student-level residual term. A similar model was used to 
analyze placement outcomes; however, this model used a logit link function 
because the outcome was binary and included only youth demographic covariates. 
The focus of all analyses was on the statistical significance and magnitude of the 
Yio coefficient which represents the impact of OTWH on caregiver outcomes. 
Statistical significance was assessed at the .05 per test alpha level. The standardized 
mean difference (e.g., Hedges’ g) between conditions was computed from model- 
adjusted means and unadjusted variances for the caregiver outcome measures. 
Odds ratios and Cox d effect sizes were computed for the binary placement 
outcomes. Odds ratios greater than 1 indicate that participants in the OTWH 
condition had higher rates of placement; values less than 1 indicate the opposite. 
Covariates were selected based primarily on theoretical justifications and to 
a lesser extent on empirical grounds (e.g., baseline equivalence). Covariates were 
used as control variables in the models. The posttest caregiver outcome models 
included three caregiver demographic variables (i.e., age, income, number of 
youths in the household) and three youth demographic variables (i.e., age, 
gender, and length of stay in TRC). The posttest and follow-up placement 
outcome models included only the three youth demographic covariates. 


Results 
Attrition and Missing Data 


Based on the observed data, the overall attrition rates for the three analytic 
samples (i.e., posttest caregiver outcomes, posttest school and community 
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placement outcomes, and follow-up school and community placement out- 
comes) were 31.55% (nm = 59), 17.11% (mn = 32), and 29.41% (n = 55), 
respectively. The differential attrition rates (between OTWH and SAU) 
were A6.60%, A6.93%, and A3.91%. 


Baseline Equivalence 


Baseline demographic variables were compared between the OTWH and 
SAU conditions (see Table 1). Effect sizes, representing the standardized 
difference between OTWH and SAU participants, were computed for each 
analytic sample. Per WWC guidelines, variables which exhibited effect sizes 
larger than 0.25 may suggest a lack of baseline equivalency between partici- 
pants in the two conditions. Of the 14 caregiver and youth demographic 
variables examined across the three analytic samples (ie. 42 total compar- 
isons), six effect sizes were greater than 0.25. Caregiver age consistently 
differed across the analytic samples with older caregivers in the OTWH 
condition. Caregiver ethnicity also differed across analytic samples with 
fewer Latino/a caregivers in the OTWH condition. 


Effectiveness 


Descriptive statistics for the four continuous posttest caregiver outcome mea- 
sures are reported in Table 2 for the OTWH and the SAU conditions, respec- 
tively. These descriptive statistics represent the unadjusted posttest means and 
standard deviations as well as the unconditional intra-class correlation (ICC) 
for participants in the OTWH condition nested within the same intervention 
cluster (i.e., nested within the same interventionist). ICCs typically range from 
0 to 1 (although negative values are possible) and represent the non-indepen- 
dence between participants in the same intervention cluster. In other words, 
the ICC is the correlation that exists between individuals in the same inter- 
vention cluster that would otherwise not exist if observations were indepen- 
dent (as in the SAU condition). ICCs can also be interpreted as the proportion 
of variance that is attributed to the cluster level. Therefore, between 8.4% (FES 
Family subscale scores) to 17.7% (FES Community subscale scores) of the 
variance in posttest caregiver outcomes is explained by the clusters. 

The main effects of OTWH were estimated using partially-nested HLM 
regression analyses where the primary focus was on the statistical significance 
and magnitude (i.e., Hedges’ g, odds ratio, Cox d) of the treatment indicator 
(y1o; the unstandardized mean difference between the OTHW and SAU 
conditions). The results of each HLM are reported in Tables 3 and 4. 
Three of the four posttest caregiver outcome measures differed significantly 
between the OTWH and SAU conditions: FES Family (y19 = 0.247, g = 0.398, 
p = .043), FES Community (y19 = 0.455, g = 0. 586, p = .001), and Caregiver 
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Table 2. Unadjusted means and standard deviations for posttest caregiver 


outcome measures. 
Outcome 
FES Family 
FES Services 
FES Community 
Self-Efficacy 


OTWH 


4.107 
4.344 
3.136 
87.868 


aa 


0.632) 
0.573) 
0.729) 
11.263) 


SAU 
3.792 
4.102 
2.695 


0.603 
0.628 


(0.603) 
(0.628) 
(0.813) 
(11.344 


82.647 (11.344) 


Table 3. Results of effectiveness analyses for posttest caregiver outcomes. 


Predictor FES Family 
Coefficients (S.E.) 
Intercept (Yoo) 3.878 (.111)* 
OTWH (y10) 0.247 (.120)* 
CG Age (y20) 0.018 (.006)* 
Low Income (y39) —0.059 (.109) 
Youths in Household (yao) 0.092 (.049) 
Youth Male (yso) —0.054 (.112) 
Youth Age (Veo) —0.006 (.037) 
LOS? (y70) 0.002 (.009) 
Variance Components 
Level 1 (0”) 0.363 
Level 2; OTWH (111) 0.009 
Exact p-value 
OTWH (y10) 043 
Effect Size (Hedges’ g) 
OTWH (y10) 0.398 


Note. * p < .05. ' = Length of stay. 


4.278 (. 
0.225 (. 
—0.001 (. 
—0.314 (. 
0.022 (.046) 
—0.052 (. 
—0.027 (, 
0.008 (: 


FES Services 


105)* 
115) 
006) 
103)* 


106) 
035) 
008) 


0.323 
0.010 


054 


0.372 


FES Community 


2.712 
0.455 
—0.007 
—0.168 
0.103 
0.096 
—0.039 
—0.004 


(.138)* 
(.135)* 
(.008) 
(135) 
(.061) 
(.138) 
(.046) 
(.011) 
0.556 
<0.001 


.001 


0.586 


ICC 


084 
117 
177 
.065 


Self-Efficacy 


2.027)* 
1.990)* 
0.111 
1.985 
0.892 
2.036 
0.672 
0.161 


85.325 
4.932 
0.052 

—3.704 
0.612 

—1.498 
0.361 

—0.006 


NN eee 


) 
) 
) 
) 
) 
) 
) 
) 


120.729 
0.115 


016 


0.434 


Table 4. Results of effectiveness analyses for posttest and follow-up placement outcomes. 


Predictor 


Coefficients (S.E.) 
Intercept (Yoo) 
OTWH (Yio) 

Youth Male (y20) 
Youth Age (y39) 
LOS (Yao) 

Variance Components 
Level 2, OTWH (u,) 

Exact p-value 
OTWH (y10) 

Effect Size (Odds Ratio) 
OTWH (yi) 

Effect Size (Cox d) 
OTWH (y10) 


Note. * p < .05. ' = Length of stay. 


Posttest Home Posttest School 


Placement 


1.412 (.368)* 
—0.003 (.384) 
—0.340 (395) 

0.155 (.135) 
—0.005 (.032) 

.001 
995 
0.997 


—0.002 


Involvement 


1.440 (.366)* 
—0.065 (374) 
—0.517 (.389) 

0.083 (.133) 
—0.006 (.031) 

.001 
862 
0.937 


—0.039 


Follow-Up 
Home 
Placement 


1.876 (.483)* 
1.114 (.512)* 
—1.131 (.550)* 
0.359 (.167)* 
—0.070 (.043) 
.010 
.033 
3.048 


0.675 


Follow-Up 
School 
Involvement 


1.069 (.369)* 
0.703 (.473) 
—0.647 (433) 
0.034 (.143) 
—0.054 (.037) 
166 
142 
2.021 


0.426 


Self-Efficacy (yi9 = 4.932, g = 0.434, p = .016). Posttest means for FES 
Services subscale did not differ significantly between participants in the two 
the conditions (19 = 0.225, g = 0.372, p = .054). For each outcome, caregivers 
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in the OTWH condition demonstrated greater levels of empowerment and 
self-efficacy compared to peers in the SAU condition. 

For the posttest placement outcomes, neither home (yi9 = —0.003, 
OR = 0.997, d = -0.002, p = .995) nor school involvement rates 
(Yio = —0.065, OR = 0.937, d = —0.039, p = .862) differed significantly 
between the OTWH and SAU conditions, and in both cases, slightly favored 
the SAU condition. For the follow-up placement outcomes, home placement 
rates differed significantly between conditions (yj9 = 1.114, OR = 3.048, 
d = 0.675, p = .033) with a larger proportion of participants in the OTWH 
condition exhibiting positive placements; however, school involvement rates 
did not differ significantly (y19 = 0.703, OR = 2.021, d = 0.426, p = .142) even 
though participants in the OTWH condition exhibited higher rates of posi- 
tive school involvement. 


Discussion 


The aims of the study were to evaluate the evidence of the efficacy of OTWH 
as an aftercare program for youth with or at-risk for emotional and beha- 
vioral disorders who transitioned back home, to their schools, and commu- 
nity after a stay in group home programs. Additionally, we evaluated the 
evidence of the long-term impact of the intervention with these youth and 
their parents. Overall, findings from this efficacy and replication study of 
OTWH were mixed. First, effects on family indicators revealed fairly con- 
sistent positive effects for families assigned to OTWH compared to families 
assigned to SAU. With the exception of FES Services, significant differences 
were found between families assigned to OTWH and SAU on FES Family, 
FES Community, and self-efficacy with more favorable outcomes for care- 
givers in OTWH. These findings suggest that OTWH is an effective program 
for improving perceptions of caregiver self-efficacy and empowerment, 
which is promising given the extensive literature supporting these factors 
as key to the long-term educational success and overall wellbeing of youths 
(Kim & Bryan, 2017; Rominov et al., 2016; Steca et al., 2011). 

Although this study did not fully replicate the findings of the previous 
RCT at 12-months (Author, 2013), both groups, OTWH and SAU, demon- 
strated home placement stability above 75%. This result is higher than other 
studies that have reported much higher reentry rates and placement stability 
of 25 - 50% in the year following discharge (McMillen, Lee, & Jonson-Reid, 
2008; Narendorf & McMillen, 2010). Additionally, at 21-months post dis- 
charge (i.e., follow-up) participants in OTWH demonstrated greater levels of 
school involvement and home placement stability. Specifically, moderate 
differences were found between OTWH and SAU participants’ school invol- 
vement rates (OR = 2.021; d = .426) suggesting that at 21-months post 
reunification, the odds of OTWH youths staying in school (i.e., attending 
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school, graduated, or working on a GED) were twice that of youths in SAU. 
Similarly, for community placement, large and significant differences 
between OTWH and SAU at follow-up were found (OR = 3.048; d = .675) 
indicating that the odds of positive placement for youths in OTWH were 
over three times greater than for youths in SAU. Given the significant 
financial, educational, and emotional costs associated with placement 
instability and educational failure (Author, 2013), these results indicate 
some promise for the long-term impact of OTWH. 

Despite evidence of promising long-term impact, the lack of replicative effects 
at posttest was disconcerting and warrants further investigation. The lack of 
replication may be related to a number of issues, but one explanation seems to 
supersede the others: that the counterfactual conditions vary between the two 
RCT studies. It is entirely likely that differences between the counterfactual 
conditions might have contributed to a lack of replication. In both the initial 
RCT study and this RCT study, the counterfactual was somewhat ill-defined. 
That is, it was not entirely clear what services participants would have experi- 
enced if not assigned to the OTWH condition. Even though the counterfactual 
condition was ill-defined in both studies, there was anecdotal evidence that 
agencies had adopted some of the OTWH framework since the completion of 
the initial RCT in 2012. Contamination of control participants can reduce the 
estimate of the effectiveness of an intervention resulting in a Type 2 error 
(Torgerson, 2001). In this study, all of the participants were provided informa- 
tion about the components of OTWH prior to randomization. Participants in 
the SAU group were also not prevented from accessing other aftercare supports. 
Since the trial was conducted in the same local area as the first study, many of the 
schools had been exposed to the school components of OTWH during the first 
study and may have had teachers/administrators who provided similar support 
to youth in the SAU group. Because some agencies in the region had adopted 
aspects of OTWH services as part of services-as-usual practices, it is possible that 
there was contamination in the SAU condition, which could account for the 
diminished effects at posttest. Future studies should collect implementation data 
from both the OTWH and SAU groups to better define the counterfactual. 

Other explanations for variation in the effects of OTWH across RCT 
studies could be related to a misspecified or incomplete OTWH logic 
model or misspecified analytic models. The findings might suggest that the 
logic model explicating the hypothesized underlying causal mechanisms 
driving the effectiveness of OTWH needs further study and refinement. 
The OTWH intervention, like many other interventions in child welfare, is 
an INUS condition (Mackie, 1965). That is, OTWH is an “insufficient but 
necessary part of a condition” for exerting positive change in the lives of 
youths leaving residential care settings. The concept of an INUS condition 
lies at the heart of the most pressing question facing intervention scientists: 
for whom, under what conditions, and through which causal mechanism(s) 
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is the intervention effective? Ultimately, it is our goal to help elucidate what 
components of OTWH are necessary and perhaps sufficient for improving 
student placement outcomes; however, this is a long process that starts with 
reevaluating the logic model underlying the intervention components. 

Relatedly, there is the potential that important “third variables” were not 
collected as part of the study or were omitted from the posttest impact 
analyses, which, if included, might have revealed moderated effects of 
OTWH. Using these RCT data, we evaluated potential moderators and 
mediators; however, these analyses did not help identify third variables that 
might be related to conditional posttest effects of OTHW and were limited 
due to sample sizes and missing data. Again, further refinement of the logic 
model and assessments of intervention uptake in both OTWH and SAU 
conditions may help further research identify potentially important contex- 
tual factors to include in moderation analyses or process factors to include in 
mediation analyses. 


Limitations and Future Research 


Several limitations should be noted and addressed in future studies. First, 
although six TRC agencies consented to participate, only three agencies 
referred youths and nearly all consenting youths (99%) discharged from the 
same large residential setting in the Midwest. One of the participating 
agencies stopped providing TRC programming shortly after agreeing to 
participate, and with the exception of the one large agency, the other agencies 
were significantly smaller in size and discharging youths rarely met inclusion 
criteria. This limitation could have contributed to the possible contamination 
and subsequent improvement in support for the SAU group given the high 
degree of exposure the staff, youth, families, and schools in this local area had 
based on the promising results from the first RCT. This also highlights the 
challenge of conducting a RCT in a real world setting (Puffer, Torgerson, & 
Watson, 2005), especially with a small and distinctive population such as 
youth departing residential programs. Given the differences in services pro- 
vided across TRCs during treatment, it is possible that these findings may not 
generalize to youths served in other residential settings. Replication is needed 
in agencies across the country that are representative of the variations in 
scope and programming provided across TRCs. 

Second, pretest measurements of the outcome variables were not collected 
as part of this RCT. This resulted in two noteworthy limitations: (1) reduc- 
tion in the statistical power to detect treatment effects, and (2) inability to 
evaluate pretest equivalence between conditions (in terms of pretest measure- 
ments). Including pretest measurements in the HLMs helps to improve the 
statistical power to detect treatment effects by explaining variance in the 
outcome that is orthogonal (i.e., uncorrelated) to the treatment indicator. If 
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pretest measurements were collected and included in the HLMs, we might 
have been able to detect a significant difference between conditions at follow- 
up for school involvement. In addition to the limitation related to statistical 
power, the lack of pretest measurements also limited our ability to evaluate 
the equivalence between conditions prior to the administration of the inter- 
vention. While random assignment should account for both observed and 
unobserved differences between conditions, attrition may introduce non- 
equivalence, so the WWC suggests evaluating the pretest equivalence of the 
analytic samples. Without establishing the pretest equivalence of the analytic 
samples, there is a possible selection threat to internal validity. 

Third, while the RCT is considered to exhibit low attrition based on overall 
and differential attrition rates, the observed rate of overall attrition was relatively 
high, which could have introduced bias into the estimates of treatment effec- 
tiveness. At the very least, attrition had deleterious effects on statistical power 
which would have limited the ability to detect differences between participants 
in the two conditions. Replication with larger samples to account for the 
challenges with collecting follow-up data on this population (e.g., in protective 
custody, high rates of mobility) and the inclusion of other mechanisms to 
promote involvement in the research is necessary to replicate findings. 

Fourth, it is possible that the Type I error rate was inflated above the 
nominal .05 level because multiple analyses were conducted; however, other 
steps were taken to limit the Type I error rate. For example, all outcomes 
were selected on a confirmatory basis and were analyzed using partially 
nested HLMs, which helped to limit the Type I error rate by accounting 
for the nesting (i.e., non-independence) of OTWH participants within inter- 
ventionists. It is also important to note that while using partially nested 
HLMs limited the Type I error rate, this approach also reduced the statistical 
power of the analyses due to the intraclass correlation in the OTWH condi- 
tion. Researchers interested in replicating this study should account for the 
partial nesting when conducting power analyses for future studies. 

Fifth, the HLMs assumed a homogeneous variance structure at level-1 (ie., 
equal variances between OTWH and SAU conditions); however, there is 
a possible heterogeneous variance structure due to the partially-nested design of 
the RCT (e.g., nesting in only the OTWH condition; Lohr et al. 2014). 
Heterogeneous variances were tested for each HLM and all of the likelihood 
ratio tests were non-significant, but the tests were likely underpowered to detect 
differences in variances. As a result, standard errors for the model may be down- 
wardly biased. In a future replication study, a larger sample of youths with more 
intervention clusters would provide greater power for detecting heterogeneous 
variances. 

Finally, HLMs for the school involvement outcomes did not include 
school-level or district-level covariates (e.g., school-wide behavior manage- 
ment approach) which might be important predictors of school involvement. 
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Researchers might consider including these types of factors in future studies. 
Furthermore, replications of this study might want to focus more on treat- 
ment effect heterogeneity (Tipton & Hedges, 2017) to understand how 
treatment effects vary across the cluster-level unit (e.g., interventionists, 
schools, states, etc.) which would allow researchers to begin to evaluate for 
whom and under what circumstances the OTWH intervention is effective. In 
this pursuit, researchers could include a greater number of interventionists to 
enable an evaluation of the degree to which the effects of OTWH are 
consistent across interventionists and to what degree interventionist charac- 
teristics (e.g., prior training, experience with OTWH, etc.) impact the effec- 
tiveness of OTWH. Researchers might also consider using a randomization 
schema that blocks randomization by school, district, or state (if possible), 
which would allow researchers to examine treatment effect variation across 
these kinds of sites. 


Conclusions 


With the passing of Family First mandating the implementation of aftercare 
for reunifying youths following placements in TRC, the need for efficacious 
aftercare programming is paramount to promoting family stability and youth 
long-term success. OTWH was developed to address the lack of rigorous 
research assessing aftercare programming and has been evaluated through 
two RCTs to determine efficacy on youth and family outcomes. This study 
extends the results of the 2012 evaluation to address long-term impact on 
youth school involvement and placement stability, while also evaluating 
program effects on key family outcomes. 

The 12-month results from the first RCT, which showed significant differ- 
ences between the OTWH and SAU, did not replicate, however the caregiver 
posttest and youth follow-up placement stability outcomes at 21 months were 
promising. While it is possible OTWH services did not work as well in 
the second study, service-as-usual improved, which could be related to con- 
tamination from the first study and explain the non-significant differences 
between the groups during the 12-month service period. Nonetheless, the 
placement stability of the OTWH group was significantly better at the 21- 
month follow-up survey compared to the SAU group. To reduce the potential 
for contamination, future studies testing OTWH should be conducted in 
a different region. The study highlights the challenge of balancing internal 
and external validity in real-world intervention studies and supports the argu- 
ment for evaluating programs through a process of repeated testing and 
refinement through small-scale experiments across multiple locations which 
may provide better evidence than larger studies (Hamilton, 2015). The study 
also provides an example of the importance of viewing program evaluation as 
a means to assist in incremental improvements that deepens the understanding 
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of programs and social policy (Cronbach et al., 1981; Gowin & Millman, 1981). 
Therefore, further rigorous replication and implementation studies are needed 
to better understand the mechanisms that most impact youth and caregiver 
outcomes at program completion and long-term follow-up. 
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